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Field of the Invention 

The present invention relates to information processing systems and observation and 
assessment methods and operations, including the training and monitoring of interviewers or 
observers, quantitative and qualitative rating and assessment processes, and the storage and 
retrieval of recordings of interviews or observed events and, more particularly, to a system 
and method for indexing recordings of interviews or other observed events or phenomena by 
combining systematic use by an observer of pre-defined descriptive or quantitative 
measurement items with simultaneous digital recording to automatically generate indexes. . 

Description of the Prior Art 

In the medical, legal, scientific, business, and education domains, among others, it is 
necessary to systematically observe and describe, assess, rate, or otherwise measure events 
or processes (i.e., phenomena). It is also useful to make recordings of these events and to be, 
able to playback these recordings. For example, physicians may use audio, video, or 
electrophysiological recording of an interview, a physical examination, a diagnostic 
procedure, an operation, or autopsy, and may need to retrieve and examine specific parts of 
that recording at some later time for various purposes. Attorneys may carry out and record a 
witness deposition and later playback sections of that oral testimony in order to link the 
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testimony to other evidence (documentary, video, or testimony by other witnesses) in order 
to draw parallels, highlight inconsistencies, or for other purposes. Social scientists apply 
systematic observation and rating methods to study society and culture. The frequency of 
such recordings, audio, video, or other, is likely to increase as their usefulness continues to 
5 grow. In general, the usefulness depends upon the purposes and value of retrieval and 

playback, the range of retrieval and playback functions, and how accurately and efficiently 
retrieval and playback can be carried out. Purposes include: providing an educational or 
teaching tool to illustrate observation, assessment, and measurement techniques in general, 
creating teaching materials; monitoring teachers, trainees, and raters; allowing for review of 

10 the assessor's conclusion or diagnoses; supporting retrieval of multiple interviews, events, or 

segments of interviews or events for comparison and other study; reviewing an interview for 
legal and/or other purposes; and other similar and varied uses in many domains. 

As a more detailed and prototypical example, medicine and public health are 
domains in which considerable effort has been devoted to the development of tools (e.g., 

15 questionnaires or assessment instruments) for conducting systematic assessment interviews 

and examinations. A main objective of these efforts has often been to improve the accuracy, 
reliability, and efficiency of a clinician's diagnosis of a patient. 

Among medical disciplines, psychiatry researchers have perhaps allocated the most 
efforts to developing patient assessment methods and tools. Examples of the types of 

20 assessment instruments that have been developed, which are composed of descriptor or 

quantitative assessment (i.e., measurement) items or questions, include the Structured 
Clinical Interview for DSM (SCID), the Diagnostic Interview Schedule (DIS), the 
Composite International Diagnostic Interview (CIDI), the World Health Organization 
(WHO) International Personality Disorders Examination (IPDE), and the WHO Schedules 

25 for Clinical Assessment in Neuropsychiatry (SCAN). 

These tests have been standardized around sets of inquiries or measurement items. 
Measurement items typically are composed of: 1) definition of a phenomenon of interest; 2) 
instructions regarding observation of the phenomenon; and 3) a quantitative rating scale. 
The scientific field of measurement, known as psychometrics, includes study of and 

30 methods pertaining to systematic and standardized observation, measurement, training, 

monitoring, and comparison. Psychometric issues of importance include efficiency, 
reliability (consistency), and validity (accuracy). 
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The response to or rating of standardized questions, inquiries, or measurement items 
and instructions is of interest, both in terms of the assessment results as well as the behavior 
manifested in the delivery of the assessment and in the response. As well as expressed 
symptoms, psychiatrists may want to measure or rate the behavioral responses of patients, or 
may wish to observe and rate the actions of other psychiatrists in training. Cardiologists 
may want to measure physical responses to changes in applied stress, such as changes in 
EKG readings during a cardiac stress test. Lawyers may seek to identify inconsistencies 
between witnesses to an accident using standardized questions. Similar examples are found 
in virtually every profession where a pattern of systematic observation and testing for 
response, and analysis of that response, is used. In all these fields, analog or digital 
recordings of audio, video, physiologic, or other aspects of observed events are used to 
assist in these processes. 

However, most anyone who has participated in retrieving and organizing interview 
or testing recorded materials for review and study realizes that the retrieval and review 
process is usually tedious, often incomplete and inaccurate, and has vast room for 
improvement. What is needed for improved retrieval and management of audio, video, or 
other recorded materials are indexes that are meaningful, accurate, and easily generated, 
manipulated, and extended. Indeed, one of the most important features of any useful 
recording and retrieval system is the ability to efficiently generate meaningful indexes for 
the recorded material. Such indexes would be especially valuable for structured or 
systematic information gathering processes in which professional or scientific standards are 
required and/or in which considerable efforts are made in training, quality control, and other 
administrative functions, e.g., a television interview based upon a list of questions, giving a 
patient a health questionnaire, a clinical or research medical assessment, conducting a 
deposition, using a list of actions to activate brain function in different ways, conducting 
focus groups, ethnographic research, or administering a graded cardiac stress test, among 
others. 

In the fields of medicine and health, the established means of recording assessments 
has generally been by use of analog audio or video tape recorders. Indexing, if used at all, 
has been generally through means such as length counters and manually added entries, or 
with hybrid combinations of tape recorders with microprocessor systems. This makes the 
process of creating indexes tedious and often incomplete. In addition, even after indexes are 



created, the use of tape recorders means that the process of searching and playback remains 
slow and inefficient. Finally, creation of generally meaningful indexes has been 
problematic. 

In the field of digital recording, many indexing schemes have been developed, 
ranging from low-level encoding to linking of text with video frames. Industry standard 
digital encoding methods such as the MPEG-2 (Society of Motion Picture Engineers) 
format, include channels for indexing based upon time code. However, most of these 
approaches generate index entries in a form not directly useful for retrieval, playback, and 
management, but rather provide basic mechanisms for supporting indexing. For example, 
one prior art system uses computer voice-to-text processing to automatically generate words 
that are then used to index the source digital audio recording. Although of some use, it does 
not provide a more organized level of indexing, since many words will be used in many 
areas of a recording, and hence do not provide specific indexing to unique areas or segments 
of a recording. For indexing to be more useful, the indexes or index items should contain 
meaningful and unique information that is associated with unique recording locations or 
segments. Many indexing methods can support this, but it is generally tedious and 
inefficient to generate such index items manually. 

Examples of prior art indexing systems include U.S. Pat. No 4,841,387, which 
discloses an indexing system using a touchpad to correlate simultaneously recorded audio or 
video information to handwritten notes made on numbered sheets of paper manually placed 
on top of the touchpad. This system is simply a means for replay of recorded audio or video 
information through manual entry of sheet numbers and physical contact with the paper in 
the approximate area of the original handwritten note, to generate a touchpad signal that is 
used to access the correlation indexed recording. The system does not anticipate correlation 
of recordings with any information other than the position of handwritten notes and a page 
number, nor does it provide any other means of identifying, searching, or retrieving 
recorded materials, nor does it support psychometric and more general observation, 
measurement, retrieval, training, monitoring, or comparison functions and operations. In the 
professional and technical arenas, it is systematic as well as quick access to specific 
passages or segments of recorded material that determines a system's usefulness, which is 
not adequately provided by utilizing the position of handwritten notes for indexing. 



European patent application publication EP 0 495 612 discloses an indexing system 
that connects manually entered computer notes to audio or video recordings, and permits the 
user to retrieve and play sections of a recording by selecting a note. The indexing is based 
upon spatial location and organization of a set of notes within a document rather than the 
meaning of the notes, requiring the user to manually associate notes by physical location 
within the electronic document. Because of this limitation, when a user wishes to associate 
a segment of the recording with an earlier note, the user must navigate to the page and 
location in the existing set of notes, presenting a serious inefficiency with lengthy notes. 
This system is also limited with regard to real-time augmentation of previously entered 
notes during recording, since the later notes will be time stamped with the time they were 
entered rather than with the time of the earlier material to which they relate or are relevant, 
lacking any automatic index linkage to the earlier material. As with U.S. Pat. No 4,841,387, 
this system requires manual entry of notes, does not standardize the process of note-taking 
nor provide any systematic means of observation, description, or assessment, does not 
combine systematic assessment with generation of indexes, does not provide a means for 
quantitative rating of observed events, and does not provide a means of retrieval and 
playback of recorded materials combined with systematic re-assessment or monitoring. 

U.S. Pat. No. 5,172,281 discloses a system that supports an operator in manually 
correlating previously recorded video material with a standard text transcript for the purpose 
of generating an index that can then be used by software that searches text and finds 
associated video indexes. This system has no capacity to operate in real-time situations. It 
does not provide means for automated definition of segments of recorded material, nor does 
it provide an automated means of generating the actual indexes. It searches the text material 
only based upon operator-entered key word comments, and is unable to take advantage of all 
the information in the original text. It is subject to the problem of specific comment words 
being used for more than one indexed timecode, requiring the operator who generates the 
indexes to keep track of and exercise judgement about the choice of comment words, a 
significant weakness because it does not provide a means of handling multiple occurrences 
of index key words in the transcript material. 

U.S. Pat. No. 4,924,387 discloses a system by which a stenographer can control a 
video camera so as to stenographically record a deposition while simultaneously operating a 
video camera and recorder. This system's main objective is to provide a system to 



accurately record a deposition, not to efficiently and precisely locate and present specific 
passages of the videotape deposition. It provides a limited indexing structure since the 
stroke record made is intended to be a verbatim transcription of the verbal testimony made 
in a courtroom, rather than any systematic observations or assessment of the proceedings in 
the courtroom. 

For all these systems described above, the user's ability to index notes or comments 
to the recording is entirely controlled through the indicia, or notes, the user has manually 
entered in a document. For both real-time and post processing, these systems require the 
user first to process mentally the recorded events and then generate and enter text to create 
notes, subjecting the system and its indexes to individual user variance in generating the text 
content. No method is suggested for facilitating or enhancing a user's personal note-taking 
style or accommodating the note-taking functions towards more standardized and systematic 
observation and assessment applications. In all these systems, no method is explicitly 
suggested for handling the case of repetition or extension of certain events or activities in a 
single recording or across multiple recordings, using a common indica for related events or 
phenomena. 

Thus, despite the number of such prior art methods, a need exists for an information 
system that will digitally record events, processes, or other observable phenomena, and 
simultaneously, efficiently, and accurately generate informative indexes in real-time, with 
minimal interference to the interviewer's or observer's ability to take notes or assess the 
flow of events. There is a more general need for systematic, structured, or standardized 
observation and assessment measurement methods, and there is a need to generate 
standardized, systematic, and quantitative indexes that are less subject to individual variance 
and which can be used between and among recorded information pertaining to different 
individuals or events so that comparisons and other analyses can be made. Furthermore, it 
would be desirable to have a system that allows subsequent annotation of an indexed digital 
recording of information in such a manner that the annotation did not disrupt the 
standardization of the index or otherwise adversely affect the ability of one set of recorded 
information in a standardized index key to be compared to another set of recorded 
information that uses the same standardized index key. 
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Objects and Advantages 

Accordingly, it is an object and advantage of the present invention to combine 
systematic, standardized observation and assessment using pre-defined descriptors or 
measurement items with simultaneous recording to automatically generate meaningful 
5 indexes for the recording in real-time. Additional objects and advantages are: 

- to provide a system and method that supports the generation and storage of new 
descriptor or measurement items with rating scales for subsequent use in systematic 
observation and assessment; 

- to provide a system and method that can import and store a set of pre-defined 
10 descriptor or measurement items, such as an existing questionnaire or set of rating scales, 

for subsequent use in systematic observation and assessment; 

- to provide a system and method that uses pre-existing descriptor or measurement 
items to structure the observation and assessment of events; 

- to provide a system and method that uses measurement items that employ 
15 quantitative rating scales; 

- to provide a system and method that efficiently creates an index for any type of 
recording by fully automatic correlation with the use of pre-existing descriptor or 
measurement items; 

- to provide a system and method that stores recorded material in digital form in 
20 centralized electronic storage; 

- to provide a system and method that stores descriptions or ratings of observed and 
assessed events or phenomena in centralized electronic storage; 

- to provide a system and method that allows unlimited repeated use of any 
particular pre-existing measurement item-in other words, an observer can repeatedly assess 

25 or rate a phenomenon or a recurring event using the same measurement item; 

- to provide a system and method that allows unlimited concatenation of indexed 
recorded material when one or more phenomena or recurring events are repeatedly assessed 
using the same measurement item; 

- to provide a system and method that can use any part or parts of the content of pre- 
30 existing descriptor or measurement items as index labels or key words; 
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- to provide a system and method in which a standard index, based upon a specific 
questionnaire or measurement instrument (i.e., a set of measurement items), can be applied 
across multiple recordings; 

- to provide a system and method that can search, sort, and/or retrieve recorded 

5 materials based upon meaningful words or combinations or words from the full text of the 

descriptor or measurement items used in the index; 

- to provide a system and method that can search, sort, and/or retrieve recorded 
materials based upon numerical ratings from standardized observation and quantitative 
assessment using measurement items with rating scales; 

10 - to provide a system and method that permits assessors to re-order recordings of 

events and compare similar events from different subjects on the basis of the associated 
measurement items in order to investigate possible relationships and/or trends not 
previously discernible; 

- to provide a system and method that allows a user, such as a student or trainee, to 
15 view and/or review prior assessment results, i.e., descriptor or measurement item assessment 

choices, by the same user or by a different user, in parallel with playback of the recording of 
the observed and assessed events or phenomena, allowing for efficient use of actual 
assessment for teaching or training; 

- to provide a system and method that allows playback of recorded materials with 
20 the prior selections of descriptors or ratings hidden from display; 

- to provide a system and method that allows re-assessment or re-rating during 
playback so that rater efficiency, accuracy, and reliability can be measured; 

- to provide a system and method that allows multiple users to make ratings based 
upon a single original recording, and that allows multiple users to playback and rate at the 

25 same time; 

- to provide a system and method that supports comparison of assessment results or 
ratings across multiple raters, subjects, events, or geographical sites, for analysis, teaching, 
monitoring assessment, or other research purposes; 

- to provide a system and method that can automatically compare ratings to make it 
30 easier to teach, monitor, and conduct research; 

- to provide a system and method that allows unlimited manual addition of index 
points and labels during playback; and 
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- to provide a system and method that allows assessors and others using a "test and 
response" method of investigation greater note-taking and annotation efficiency and 
flexibility than was previously available. 

Further objects and advantages of the invention will become apparent from a 
consideration of the drawings and ensuing discussion. 

Summary of the Invention 

The present invention combines systematic observation and assessment using pre- 
defined descriptor or measurement questions or items with simultaneous digital recording in 
order to automatically generate meaningful indexes that support improvements in efficiency, 
consistency, and accuracy in retrieving and utilizing the recorded material. Based upon the 
contents of the measurement questions or items, including any qualitative or quantitative 
descriptions or numerical rating results from use of the descriptors or measurement items, 
the indexes can readily be further processed and manipulated to additional advantage. The 
invention provides a method for access and manipulation of digital recordings of events by 
use of these pre-entered or pre-existing measurement items as meaningful index information 
designators such as key words, labels, item numbers, or numerical ratings. Such a method is 
particularly useful for processes relating to systematic interview or assessment methods, 
including the training and monitoring of interviewers or assessors and the storage, retrieval, 
analysis, and other manipulation of recordings. 

Brief Description of the Drawings 

In the drawings, closely related figures have the same number but different 
alphabetic suffixes. 

Figs. 1 A and IB show an example of a measurement item from the World Health 
Organization Schedules for Clinical Assessment in Neuropsychiatry. 

Fig. 2 is a generalized block diagram illustrating the configuration of the systematic 
assessment and simultaneous recording and automatic indexing functions of a preferred 
embodiment of the present invention. 

Figs. 3A and 3B illustrate further details of the systematic assessment and 
simultaneous recording and automatic indexing functions of a preferred embodiment of the 
present invention. 



Fig. 4 is a block diagram illustrating the configuration of the search, sort, retrieval, . 
playback, and manipulation functions of a preferred embodiment of the present invention. 

Fig. 5 is a block diagram illustrating the configuration of post-processing systematic 
re-assessment functions of a preferred embodiment of the present invention. 

Figs. 6A and 6B show screen shots of a preferred embodiment of the present 
invention. 

Detailed Description 

The present invention is a means and method for automatically generating 
meaningful indexes for recorded material. In the following detailed description, numerous 
specific details are set forth in order to provide a thorough understanding of the present 
invention. However, it will be apparent to one of ordinary skill in the art that these specific 
details need not be used to practice the present invention. In other instances, well known 
structures, interfaces, and processes have not been shown in detail in order not to 
unnecessarily obscure the present invention. 

A preferred embodiment of the invention will now be described herein below with 
reference to the drawings. To explain clearly, explanation will be made in accordance with 
the following order: 

(1) with respect to conducting an assessment interview; 

(2) with respect to playing back an indexed interview; 

(3) with respect to searching and retrieving indexed materials; 

(4) with respect to re-assessing a prior assessment interview. 
(1) Conducting an assessment interview 

Figures 1 A (item) and IB (rating scale) illustrate a typical measurement item from 
the World Health Organization (WHO) Schedules for Clinical Assessment in 
Neuropsychiatry (SCAN). In general, measurement items typically are composed of: 1) 
definition of a phenomenon of interest; 2) instructions regarding observation of the 
phenomenon; and 3) a quantitative rating scale. Suitable measurement items can vary over 
a wide range and will depend on the information sought from the test subject and the type of 
analysis intended. Examples of analysis include behavioral responses to audio and/or visual 
information (questions, directions, lights, sounds, etc.), physical examinations of laboratory 
specimens, oral depositions of witnesses, etc. The SCAN represents a detailed instrument 
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for clinically-based, standardized psychiatric assessment of psychopathology phenomena. 
The SCAN contains over 1 5 500 measurement items, each of which is composed of at least 
an item number 102, an item title 104, and an item rating scale 120 (Fig. IB) with rating 
scale text 122 and rating scale quantitative ratings 124, and optionally an item definition 
106, an item glossary entry 108 providing further definition, suggested probes 110 
(questions) for interview use, and other instructions 112, all of which combined are referred 
to as the "content" of a SCAN measurement item. Each measurement item as a whole with 
its constituent content may be considered as an object with respect to its function for 
conducting an assessment interview and indexing a simultaneous recording. 

Figure 2 shows a generalized block diagram of a preferred embodiment of the 
present invention. In this typical architecture, the system display presents a set of pre- 
defined measurement items 202 to the user, while simultaneously processing and digitally 
recording the observable event to be rated, in this case the psychiatric interview 204. As the 
user carries out systematic observation and assessment with selection of a measurement item 
206, optionally generating measurement result data by using the rating scale of the 
measurement item, the system automatically defines a segment of recorded material 208, 
based upon the repeated 214 selection by the user of measurement items from the displayed 
set over the course of observation and assessment. The end of the previous segment and the 
start of the next segment is defined when a measurement item is selected. For the first 
segment, no prior segment end is defined, and the start of the segment is defined by the start 
of the digital recording. Each segment is indexed to the currently selected measurement 
item that defines the start of the segment 210. The combined segment definition and 
segment indexing functions 212 are carried out in the background without user observation. 
The digital recording, measurement result data, and index information are stored in a data 
base 220. The index information thus includes the text content 102, 104, 106, 108, 110, 
112, and 120, of each associated measurement item and the rating scale 120 results 122 and 
124, which can be used as a common set of key words, labels, and quantitative attributes to 
identify index entries and allow retrieval of diverse recorded event data from different 
subjects under a common framework. The resulting assessment forms a data base file that 
can be stored, shared, or integrated into other digital records as needed by the user. 

Figures 3A and 3B illustrate further details of the systematic assessment and 
simultaneous recording and automatic indexing functions of a preferred embodiment of the 
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present invention. The system provides the user with a set of pre-defined measurement 
items and means for navigation and selection 302. Simultaneously, a process of digital 
recording occurs 304. When a measurement item is selected 306, the system automatically 
gets lower-level address information such as a time code to identify the start of the 
recording segment 308, and links the segment start address with the current measurement 
item 310. 

While these addressing and linking processes occur in the background, the user 
assesses the subject, optionally choosing to use the rating scale from the current 
measurement item to generate measurement rating data 312. When the next measurement 
item is selected by the user 314, the system automatically gets lower-level address 
information to identify the end of the recording segment 316, creates an index item by 
linking recording segment start and end identifiers to the employed measurement item, 
along with any measurement results data 320, and then stores the index item in the data base 
220. If the measurement item was previously used, and therefor there is an existing index 
item 324 using that measurement item, then the recording segment start and end identifiers 
are appended 328 to the existing index item and stored in the data base 220. 

Figure 4 is a block diagram illustrating the configuration of the query, retrieval, 
manipulation, playback, and indexing functions of a preferred embodiment of the present 
invention. One means of retrieving recorded material is through use of the pre-defined 
measurement items 402 that were used in the creation of the index items. From the display, 
the user can view and select a measurement item. Likewise, the data base can be queried by 
searching the index items for key words from the content of the associated measurement 
items 404. The data base can also be queried by searching for specified assessment results 
406. From these actions, either singly or combined 408, the system automatically generates 
a query 410 to the data base 220 that retrieves 412 the associated segment or segments of 
recorded material for further management 416. Any search results can be sorted or filtered 
using standard means, and can be generalized to search across multiple subjects and 
multiple assessments using standard means. Retrieved recorded material can be selected 
and manipulated for output to another system or file, or can be played back with standard 
functions 418. During playback, additional indexing can be performed. If further indexing 
is desired by the user 420, the system supports a choice 422 of creating and using new index 
labels 424 or using existing measurement items 426. The indexing process 428 described 
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earlier associates these index elements with recorded material, and the results are stored 430 
in the data base 220. 

Figure 5 is illustrates the configuration of post-processing systematic re-assessment 
functions of a preferred embodiment of the present invention. A re-assessment mode 502, 
re-assessment subject 504, and an existing assessment 506 can be selected 502, allowing the 
observation through playback of the recorded materials association with the existing 
assessment 506. The set of pre-defined measurement that was used to carry out the original 
assessment is displayed 508, allowing its use for navigation and selection of specific 
indexed segments of recorded materials. Standard playback functions are also available 
512. The materials observed through navigation and playback can then be re-assessed using 
the set of pre-defined measurement items 514, and the re-assessment results stored 516 in 
the data base 220. Standard management functions can then be used to process the stored 
data 520. 

Figures 6A (assessment) and 6B (playback) illustrate a preferred embodiment of the 
present invention as implemented in a working software program ("CliniMetrica"). For 
assessment, a set of pre-defined measurement items is displayed 602, from which the user 
can select the current measurement item 604. The rating scale text 608 and quantitative 
values 606 allow systematic rating. Digital audio and video recording can be started 620, 
and will be automatically indexed in sequence with the selected measurement items. For 
playback, the list of measurement items used in the assessment is displayed 632, allowing 
retrieval of the correlated segment of recorded material, for playback via a standard set of 
playback controls 640. Further, the indexed recorded material can be retrieved through text 
search of all of the measurement item, including rating scale, text 638. The recorded 
material that is played back can be re-assessed 650. 

The invention is described in detail with reference to inquiries from an assessor and 
responses made by an interviewee during an oral interview assessment. It will be 
understood that the description is equally applicable to any form of systematic assessment or 
observation by verbal, visual, audio, or physical method (e.g., questions or directions 
whether presented orally or in text form, mental or physical tests or testing, use of software 
or hardware, etc.) and any form of response by the subject, human or other, of observation 
(verbal, nonverbal, or physical response) with mere changes in semantics to describe the 
particular form of test and response. 
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A general testing system according to the invention includes the following steps: (a) 
presenting a set of pre-defined measurement items to a subject or operator for use in 
structuring and carrying out the observation and assessment of events or phenomena of 
interest; (b) simultaneously making a digital recording of one or more aspects of the 
observed events or phenomena on a digital storage medium, with said digital recording 
including an event track of event data sequenced by a time track of time data; (c) 
automatically indexing said digital recording by automatically entering time data and an 
identifier relating to the current selected measurement item into a record in a computer data 
base; and (d) optionally, augmenting said computer data base by adding information to said 
data base that relates to said measurement items or said digital recording without changing 
previously established indexing relationships and data base records. 

An interviewing assessment system according to the invention includes a computer 
having: (a) an output communication subsystem that allows communication with an 
interviewer or interviewee so that pre-defined measurement items can be presented and 
selected (b) an input subsystem for the interviewer or interviewee to respond to the 
measurement item communications to generate assessment data results; (c) a recording 
subsystem that simultaneously makes a digital record of the audio, video, physiologic, or 
other types of recordable event data from either or both of the interviewer's and 
interviewee's behaviors and responses, with a data track of said event data synchronized to a 
time track of time data; (d) an indexing subsystem that automatically generates correlation 
entries that define time segments of the digital recording based upon the selection or 
presentation of successive measurement items, and links those time segments to the 
measurement items, thereby creating a meaningful index for these segments of digital 
recording by use of the contents of the correlated measurement items and any quantitative 
measurement rating results; and (e) a digital storage retrieval subsystem that uses the 
measurement item-based index to retrieve segments of recorded material for playback, 
analysis, editing, annotating, re-rating, or other related purposes. 

The means for presenting the subject or operator with measurement items can take 
many forms, depending on the type of measurement item involved with the assessment. The 
system and its method of use is described here in the context of an interview between a 
subject and an interviewer using a computer to display a series of measurement item 
questions to the interviewer as a guide for the interview. At the same time, the interview is 
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recorded by a audio or video recorder (preferably digital) that includes a data track 
synchronized by a time track of time data. As the interviewer moves from one question 
(i.e., measurement item) to a new question via input into the computer, this question- 
switching input is an act that causes the data base (either from software or hardware 
functions) to record the time data (preferably a stop data bit for termination of the preceding 
time segment of recording and a start data bit for beginning the next time segment of 
recording) and an associated identifier that links the time segment of recording to the new 
question (measurement item), thereby generating an index entry. 

As those skilled in the art will appreciate, the indexing act can be done in an active 
interview situation, by the interviewer, as questions are posed. The indexing can also be 
done in passive situation, where the operator is observing but not necessarily actively 
interacting, by the operator entering inputs into the computer from a list of predefined topic 
items (measurement items) during actions by the subject of observation. Either method will 
generate a data base of event data that is indexed by a series of measurement item-based 
identifiers that can be sorted and retrieved for selective retrieval and playback of the events. 

The preferred software embodiment of the invention for a personal computer will 
have the capability to: (i) communicate pre-defined inquiries to the interviewee directly or 
through an interviewer in a systematic manner; (ii) provide rating scales or methods 
associated with or part of the pre-defined inquiries by which said inquiries can be rated or 
applied for the purpose of describing said interviewee; and (iii) automatically generate an 
index in the form of retrievable index marks associated with the pre-defined inquiries. A 
personal computer is used to communicate the inquiries and at the same time capture and 
digitally record or control a recording device that records the interviewee's responses to 
form a digital record of inquiries and indexed responses. As those skilled in the art will 
appreciate, any type of computing device could be used to provide these functions, including 
customized or programmable hardware that provides functions described above as provided 
by software. 

If digitally recorded responses are to be further indexed during later playback, the 
system can also insert additional index marks and the identifiers for the index marks into the 
digital record so that digital materials are further indexed for retrieval. For example, in 
reviewing an earlier interview, an assessor may wish to employ pre-defined measurement 
items not used originally in interview, to further assess or rate the interview. In addition to 
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use of pre-defined measurement items which provide meaningful text content that serves as 
index labels or key words, new index marks can be placed at any point in the recording with 
manual entry of a label or key words. Furthermore, if the user wishes for example to study 
or analyze communication and response patterns, new measurement items could be created 
5 and stored for repeated use. Preferably, an "inquiry" measurement item could be created, 

stored, and then inserted at each location of interest where a specified data item inquiry is 
being communicated, and a "response" measurement item could be created, stored, and 
inserted where the response is made. The inquiry and response index marks are desirably 
standardized and relate to the systematic content of the inquiry or the response. 

10 The invention is conveniently explained in the context of an interview under the 

WHO Schedules for Clinical Assessment in Neuropsychiatry (SCAN). These schedules 
represent a detailed instrument for clinically-based, standardized psychiatric assessment of 
psychopathology phenomena. The SCAN contains over 1,500 measurement items, each of 
which is composed of at least an item number, an item title, and an item rating scale, and 

15 optionally an item definition, an item glossary entry providing further definition, suggested 

probes (questions) for interview use, and other instructions, all of which combined are 
referred to as the "content" of a SCAN measurement item. An example measurement item 
from the SCAN is shown in Figs. 1 A and IB. The SCAN is a widely recognized and used 
international standard, having been translated into over 25 languages. It will be understood 

20 by those skilled in the art that the techniques applied to a SCAN assessment would be 

applicable to other types of assessment instruments, physical examinations, or other 
physical phenomenon whose analysis is aided with a recording of sights, sounds, 
physiology, and/or other domains of observation with only minor semantic differences that 
describe the same essential types of events. 

25 Using the present invention to conduct a SCAN psychiatric assessment interview 

with a human subject, real-time selection by the interviewer of a relevant SCAN item in the 
natural course of the interview, observation, and assessment of clinical phenomena 
automatically generates a time-based index entry link that allows use of the SCAN item as 
an informative indicum for the simultaneously recorded data (e.g., any of audio, video, 

30 physiologic). The time-stamp index marks (time stamp plus identifier) are stored in data 

base fields with time data that is correlated to a sequence of event data in the recording of 
the events. The resultant set of time stamp index marks then provides a general index 
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structure which can be used to further support, independently of the real-time process, a 
flexible note-taking structure and user interface that complements diverse personal note- 
taking styles and application needs. 

Notations and annotations in the form of text, sound, video, or other data can be 
5 entered into the data base as separate fields of information without affecting the relationship 

between the time stamp index mark and the recorded data associated therewith. Thus, the 
use of digital index marks of an event recording in an editable data base provides assessors 
with the ability to generate assessment records rich in diverse content. When coupled with 
standardized assessment inquiries (e.g., a list of standard interview questions) and a data 
10 base field that identifies each inquiry by a standardized descriptor for each inquiry, this 

assessment record can be used as a basis for comprehensive comparisons on diverse 
content. 

In the case of assessment interviews in which there is no pre-existing set of 
standardized measurement items (questions, definitions, inquiries, instructions, directions, 

15 or other tasks), an important feature of the present invention is the ability for the assessor or 

another individual (e.g., an interview designer) to generate a set of items prior to the event 
to be observed, assessed, and recorded. This supports the process of developing a new set of 
standardized inquiries for investigating new issues. This set of items can be used and re- 
used as real-time index information designators which can later be used as standard access 

20 mechanisms into both the notes and the correlated recorded signals. These new inquiries, 

e.g., key words, labels, or any other pointer to information, can identify specific concepts in 
the user-produced information to which the user wants to have later access. An information 
designator is treated as an object that can be easily created and reused. This novel 
implementation both in the user interface and data structure of the present invention 

25 provides a flexible and powerful indexing tool for the recorded signals in addition to the 

access provided by the recorded medium's address parameters. 

In an environment where a recording of events is being made simultaneously with a 
the rating of those events, the interviewer has limitations, due to limitations of human 
cognitive abilities, on what actions can be taken or what additional items can be considered 

30 as the interview is in progress. The present invention produces a digital data base of 

indexed inquiries and responses that can be used to organize the assessor's notes, originally 
written in free text or graphical form, but later entered into the data base with minimal 
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restriction as to the amount of that information or when it is entered. The inquiry and/or 
response index marks allow the assessor's notes to be correlated with the recorded signals of 
the event. Further, the present invention provides the assessor with the ability to modify any 
notes that were previously entered without regard to when the modifications are made or 
where they are located on the recording medium. 

The invention recognizes, moreover, that the temporal or spatial sequencing of 
information is itself often a significant clue as to the content of the information and may aid 
a user in recalling memories or in retrieving recorded information from a perceived event. 
The inquiry and response index marks of the present invention provide a mechanism for the 
user to control the playback sequence and thereby view the index with items sorted in, e.g., 
chronological order, type of inquiry (as denoted by the type of inquiry index encoded into 
the digital record), by type of response (also as denoted by the type of response index mark 
encoded into the digital record), as well as any other order desired by the assessor. 

The system of the present invention is particularly effective for video and audio 
interviews where the events include a time sequence of responses by the interviewee or test 
subject. These events are recorded as a series of audio or video input data segments 
sequenced by a time track of time data. Because the amount of detail that may be captured 
for a particular time on an associated recorded medium is open-ended, a richer set of 
annotations may be efficiently made by use of pre-defined measurement items while the 
video is being digitally recorded in real-time than is typically allowed in previous analog 
systems. This relieves the note-taker (i.e. the measurement assessor) of many of the 
pressures typically associated with taking interview notes in real time. The time sequence 
and reusable information designator object features provide for treating the event as a whole 
and permit later augmentation or modification of previously made notes. 

It is another significant advantage that the system of the present invention operates 
in both real-time and post-processing environments. This feature recognizes that note- 
taking as a human activity is typically an interactive process. The output data structure 
(e.g., the indexed video materials) produced by the system of the present invention during a 
real-time note-taking session while the event is being recorded may be further modified 
and/or augmented by the same user or by different users during post-processing review 
sessions occurring in parallel or serial. This facilitates and potentially reduces the time 
required in post-processing sessions. 
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A system for correlating event data being recorded in real time to event information 
entered into the system by a system user includes input circuitry connected to a user input 
device for producing signals indicating actions of a system user; output circuitry connected 
to a display having area for presenting images; and a processor connected for receiving the 
signals from the input circuitry, and connected for providing images to the output circuitry. 
The system also includes a digital recording device in communication with a digital storage 
medium for producing a plurality of signals indicating recorded event data about an event 
for storage on the storage medium. The recording device is currently producing the 
plurality of signals and indicating the recorded event data and is connected for 
simultaneously providing address data items to the processor of the system. The recording 
device stores the plurality of signals indicating the recorded event data on the storage 
medium so that a recording address value of each address data item provided by the address 
source indicates a portion of the recorded event data. The system also includes memory for 
storing data, including instruction data that indicate instructions the processor executes. 

Stated another way, the system may be configured to operate in a post-processing 
environment so that an event that has already been recorded but not indexed is the event 
about which a log is constructed according to the present invention. The event that has 
already been recorded is digitally stored on a storage medium as a plurality of signals 
indicating recorded event data about the event. In this aspect of the invention, the system 
includes an output device for presenting the previously recorded signals thereon to the 
system user. This output device is in communication with the storage medium including the 
plurality of signals indicating recorded event data about the event. The output device is 
connected for providing a plurality of address data items to the processor of the system as 
user-requested address values; each address data item is a recording address value indicating 
a portion of the recorded event data. Preferably, the plurality of signals indicating the 
recorded event data includes the plurality of address data items, but the output device may 
itself generate them as the plurality of signals is being presented by the output device. 

The system of the present invention permits a system user to interactively produce 
stored information that is correlated with an "address". The "address" refers to a sequence 
dimension used to identify a portion of the stored information. When the stored information 
is being correlated with recorded signals, the address typically refers to the location of the 
recorded signals on the storage medium storing the recorded signals. However, the system 
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of the present information is not limited to correlating information with recorded signals, 
and may be used in any situation where later access to the stored information would be 
facilitated by correlation with an address, and by having an access mechanism as provided 
by the "information designators". Generally, when the system user is producing a data base 
of stored information, the address record will be time data taken from the time track of the 
recording of the events. 

The computer-controlled system of the present invention uses common, readily 
available hardware components. For an assessment interview delivered via computer, the 
system includes a computer with an image display that will display questions or directions, 
an input that receives signals from a user interaction device, e.g., keyboard, digital tablet, 
microphone, camera, or other recording device that produces a data track synchronized by a 
time track. The display can present the measurement item questions directly to the subject 
or, alternatively, can display a series of questions to an interviewer/assessor who then 
presents the questions to the subject. Some act taken to move from one question to another 
generates the data base time entry or start/stop time entries as well as the index identifier 
that pertains to the question. 

Analog recording devices can have their recordings digitized, or the observable 
events can be recorded directly as a digital recording that includes: (a) a data track of sound, 
image, or other event data that are synchronized by (b) a time track that includes time data 
correlated with event data. The use and existence of these two data tracks is typically 
transparent to the user of the equipment but can be read by a computer and used to generate 
data base entries for each recorded event based on the time track data associated therewith 
from the digital recording. 

The user interaction devices useful in the present invention may include any one of a 
variety of input devices controllable by a human user. For example, the user interaction 
device may include a pointing and gesturing device to indicate actions, gestures, and strokes 
such as those produced when making handwriting gestures. A stylus or other form of 
handheld pen are examples of a suitable user interaction device. Other pointing devices 
such as a mouse, trackball, eye position, or brain activity electrical or magnetic patterns may 
also be suitable for indicating actions, gestures, and strokes. 

Although the foregoing disclosure relates to preferred embodiments of the invention, 
it is understood that these details have been given for the purposes of clarification only. 
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Various changes and modifications of the invention will be apparent, to one having ordinary 
skill in the art, without departing from the spirit and scope of the invention. 



21 



